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BACKGROUND OF THE INVENTION 

1 . Field of the Invention. 

The present invention relates generally to interacting with video, animation, or a 
sequence of frames on a computer, and in particular, to a method, apparatus, and article 
5 of manufacture for annotating video, animation, or a sequence of frames on a computer. 

2. Description of the Related Art. 

Computer users, developers, and programmers may commonly attach 
annotations to static images of designs, word processing documents, etc. Annotations 

10 may include redlines, text, images, markup data, notes, a box, a circle, an ellipse, a spline, 
a polyline, a group, an arc, a cloud, a callout, a video (e.g., a video chp), an audio 
recording (e.g., an audio clip), or any other object/entity tiiat may be used to comment 
or markup. However, such functionality is not available with respect to video, 
animation, or a sequence of images displayed on a computer system. 

15 While video clips and animation may be displayed using a computer, the prior art 

fails to provide a mechanism for a user to markup or comment on the video cUp, and/or 
a sequence of images. Further, the pnor art fails to provide a satisfactory method for 
creating, transmitting, and using such annotations. 

20 SUMMARY OF THE INVENTION 

One or more embodiments of the invention provide the ability to annotate 
video, animation, or a sequence of frames using a computer. Using embodiments of the 
invention, users may be provided with the abilit}^ to create, transmit, and use such an 
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annotation. One or more advantages that may be available pursuant to the invention 
include the ability to annotate or provide instructions for the assembly of a product, 
point out a notable moment in a movie to a friend, comment and illustrate on how a 
product is operating, or comment and illustrate how a service is being performed. 
5 Annotation information is obtained from a user or application that specifies an 

annotation, the frame to be annotated, and the location on the frame where the 
annotation is to be displayed. A sequence of frames such as a video clip or animation is 
then displayed. When the indicated frame is displayed, the display process is paused and 
the annotation is displayed/overlaid on the frame at the location specified. 
10 The annotation information may be defined in accordance with an extensible 

markup language (XML) schema and is likely stored as a separate document/object from 
the sequence of firames that are being annotated. Since the annotation information is 
stored separately, it is likely a small document that may be easily and quickly transmitted 
across a network. 

15 

BRIEF DESCRIPTION OF THE DEAWINGS 
Referring now to the drawings in which like reference nvimbers represent 
corresponding parts throughout: 

FIG. 1 is an exemplary hardware and software environment used to implement 
20 one or more embodiments of the invention: 

FIG, 2 is a flow chart illustrating the display of an annotation in accordance with 
one or more embodiments of the invention; and 

FIGS. 3A-3E ate frames of an animation sequence that illustrates the assembly 
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of a connecting rod in accordance with one or more embodiments of the invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
In the foRowing description, reference is made to the accompanying drawings 
which form a part hereof, and which is shown, by way of illustration, several 
embodiments of the present invention. It is understood that other embodiments may be 
5 utilized and structural changes may be made without departing from the scope of the 
present invention. 

Overview 

Annotation information is obtained that specifies an annotation (or type of 
10 annotation), a frame to display the annotation on, and a location that specifies where on 
the frame the annotation is to be displayed. When a sequence of frames is displayed, the 
display sequence is paused on the specified frame and the annotation is 
displayed/ overlaid on the frame at the specified location. 

15 Hardware Environment 

FIG. 1 is an exemplary hardware and software environment used to implement 
one or more embodiments of the invention. Embodiments of the invention are typically 
implemented using a computer 100, which generally includes, inter alia, a display device 
102 (such as a monitor), data storage devices 104, cursor control devices 106, and other 

20 devices. Those skilled in the art wiU recognize that any combination of the above 

components, or any number of different components, peripherals, and other devices, 
may be used with the computer 100. Further, embodiments of the invention may be 
implemented on a handheld device such as a PaknPilot or Windows CE device that has 
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a display device 102. 

One or more embodiments of the invention are implemented by a frame 
annotation program 108, wherein the firame annotation program 108 is represented by a 
window displayed on the display device 102. Generally, the frame annotation program 
5 108 comprises logic and/ or data embodied in or readable from a device, media, carrier, 
or signal, e.g., one or more fixed and/ or removable data storage devices 104 connected 
direcdy or indirecdy to the computer 100, one or more remote devices coupled to the 
computer 100 via a data communications device, etc. 

Those skilled in the art will recogni2e that the exemplary environment illustrated 
10 in FIG. 1 is not intended to limit the present invention. Indeed, those skilled in the art 
will recogni2e that other alternative environments may be used without departing from 
the scope of the present invention. 



Software Embodiments 

15 In accordance with one or more embodiments of the invention, frame 

annotation program 108 provides the ability to display a sequence of images, a video, or 
animation on display device 102. Such a sequence of images may be in a variety of 
formats including Autodesk Animation, AVI (audio video interleaved), MPEG (moving 
picture expert group), QuickTime, RIFF (resource interchange file format), animated 

20 GIFs (graphic interchange format), ASF (active/advanced streaming format), smacker, 
RealMedia streaming format, Vivo's H.263, Iterated Systems/Progressive 
Networks/RealNetworks RealVideo/ClearVideo, VDOWave, VxTreme, Duck 
TrueMotion, or a sequence of stiU images in separate files. However, embodiments of 
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the invention may be utilized with any type of available format. 

The frame annotation program 108 maintains the ability to pause the sequence 
of images on a particular frame/image and display/overlay an annotation on the frame. 
Alternatively, instead of displaying/ overlaying an annotation on the frame, frame 
5 annotation program 108 may play an audio clip, separate video clip, or other multimedia 
on display device 102. 

The annotation may be displayed on a specified location on the paused frame. 
To provide such capabilities, annotation information is provided to application 108. The 
annotation information may include the identification of the frame, an annotation, and a 

10 location on the identified frame to display the annotation. As described above, an 
annotation may include redlines, text, images, markup data, notes, a box, a circle, an 
ellipse, a spline, a polyline, a group, an arc, a cloud, a callout, a video (e.g., a video cUp), 
an audio recording (e.g., an audio clip), or any other object/entity that may be used to 
comment or markup. Thus, the annotation may be a prrtnitive shape or a complex 

15 shape. 

The annotation and location may be integrated such that the location specifies a 
series of points or lines that comprise a line, an arrow, or other object. Alternatively, 
instead of specifying a location, a default location may be assumed or used. For 
example, if die annotation comprises text, the appKcation 108 may display the text at a 
20 default location such as across the top of the frame. 

Additionally, die annotation information may be stored in a file separate from 
die sequence of images (e.g., the AVI file) diat it is associated with. By storing the file 
separately, the annotation information may be quickly transmitted, transferred, etc. 
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The annotation information may be defined in accordance with an extensible 

markup language (XML) schema. Using XML allows the markup/annotation data to be 

transmitted over standard Internet connections. In the XML data, a tag reference to the 

sequence of frames associated with the annotation and a frame location for where the 

5 markup annotation should appear are provided. The following is an example of the 

implementation/use of an annotation implemented using an XML document. 

<Request action="SavePageXML" Pageld="15"> 
<MarkUp> 

<Objects> 

10 <Arrow> 

<Line Weight="1" Style="0" Color="255" /> 
<Point2d x="333." y="194." /> 
<Point2d x="401 y="278." /> 
<Point2d x="401 y="278." /> 
1 5 <Point2d x="389." y="268." /> 

<Point2d x="393." y="264." /> 
</Arrow> 
</Objects> 
</MarkUp> 
20 <Comments /> 

<Viewers> 

<Viewer type="Windows Media" ref="Forensic/animation1,avi" 
sframe="54" /> 

<A/iewers> 
25 </Request> 

The example above is from a frame annotation program 108 where a user has 
placed an image of an arrow (see </Arrow> element) (e.g., using a cursor control device 
106) that corresponds to frame 54 of an AVI file (<Viewer type="Windows Media" 

30 ref="Forensic/aninnation1 .avi" sframe="54" />). The line weight (<Line Weight="1 " Styie="0" 
Color="255" /> ) and points (<Point2d x="" y="" />) tiiat are part of tiie arrow element 
specify/define die points of die arrow. After loading die above XML document, the 
frame annotation program 108 will pause the display of die AVI file called 
"Forensic/animationl.avi" at frame 54 and display the arrow defined by the points 

35 specified. 
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If an annotation other than an arrow were to be utilized, a tag for the element 
would be provided along with information as to the location of the annotation within 
the element. For example, a circle annotation may appear as follows: 

<Circle> 

<Line Weight="1" Style="0" CoIor="255" /> 
<Point2d x="333," y="194." /> 
<Radius="5" /> 

</Circle> 

Thus, depending on the type of annotation, different elements may be specified/utilized 
(e.g., a radius for a circle, text, an equation for a parabola, etc.). Additionally, in the 
above example, if multiple annotations are used, each annotation may be listed as a new 
element/ object within the <Objects> and </Objects> tags. Annotations may be 
created/ defined by a user (e.g., a person interacting/viewing the sequence of fcames) or 
may be created/ defined by any other party (e.g., the creator of the frame sequence), 

FIG. 2 is a flow chart illustrating the display of an annotation in accordance with 
one or more embodiments of the invention. At step 202, a sequence of frames is 
obtained. At step 204, annotation information is obtained. As indicated above, 
annotation information may include an identification of a ftame, an annotation, and a 
location on the identified frame to display the annotation. At step 206, a frame from the 
sequence of frames is displayed. 

At step 208, a determination is made regarding whether the frame is the 
identified frame. If not, the process continues and the next frame is displayed at step 
206. If the frame is the identified frame, frame annotation program 108 pauses in 
displaying the sequence of frames at step 210. At step 212, the annotation is 
displayed/played at die location specified. Thereafter, the sequence of frames may 
continue to be displayed upon the user electing to proceed (e.g., by selectiing a "Play" 




button ot other key on a keyboard ot input device that acts to un-pause the frame 
sequencing). 

Specific Examples 

5 The above described embodiments may be utilized in a variety of situations. For 

example, FIGS. 3A-3E illustrate frames in an animation sequence for the assembly of a 
connecting rod. In the animation sequence, the end caps 302 move towards connecting 
rod 304 and are secured to connecting rod 304 using nuts 306 and bolts 308. FIGS. 3A- 
3E are each single frames in the sequence. FIG. 3C illustrates an annotation 310 of a 

10 frame in the sequence. Thus, during die animation sequence, the animation is paused on 
the frame of FIG. 3C and annotation 310 appears. In FIG. 3C, annotation 310 is a note 
with an arrow to a bolt 308 that reminds the user to torque the bolts 308 to a specified 
level (e.g., 75 ft.-lbs.). The user may then elect to continue the animation (e.g., by 
pressing a "Play" button), the annotation disappears, and the animation continues with 

15 FIGS. 3D and 3E. 

In another example, a consumer may purchase a bookshelf from a store and the 
consumer has to assemble the bookshelf at home. Before leaving the store, the 
consimier may download the assembly instructions onto the computer 100. The 
instructions guide the user through the assembly with an animated example. At crucial 

20 points in the process, the animation may stop and display additional annotations to 
further explain the process. The consumer may also add their own notes and 
annotations to die instructions to be used the next time the item is assembled or 
disassembled. 
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Conclusion 

This concludes the description of the preferred embodiment of the invention. 
The following describes some alternative embodiments for accomplishing the present 
5 invention. For example, any type of computer, such as a mainframe, minicomputer, or 
personal computer, or computer configuration, such as a timesharing mainframe, local 
area network, standalone personal computer, Windows CE device, PalmPilot, or 
handheld computers, could be used with the present invention. In summary, 
embodiments of the invention provide a method for annotating a sequence of frames. A 

10 frame in a sequence, an annotation, and a location on the frame for the annotation is 
identified and loaded into a frame annotation program 108. The sequence of frames is 
then displayed. When the identified frame is displayed, the display sequence is paused 
and the annotation is displayed at the identified location. 

The foregoing description of the preferred embodiment of the invention has 

1 5 been presented for the purposes of illustration and description. It is not intended to be 
exhaustive or to limit the invention to the precise form disclosed. Many modifications 
and variations are possible in light of the above teaching. It is intended that the scope of 
the invention be limited not by this detailed description, but rather by the claims 
appended hereto. 
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